gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
6823 commits 28 branches 278 tags 135 MiB
Commit graph

592 commits

Author SHA1 Message Date
Chris Lu d9a2a7f1c4 WIP 
no memory issue

if some directory is removed, it may have this error

$ rm -Rf ~/tmp/m2/s1
rm: fts_read: Device not configured
 2021-04-18 13:06:38 -07:00
Chris Lu 54410ca955 cleaner way to set readonly 2021-04-18 10:02:02 -07:00
Chris Lu c83ab91e2e remove unused variable 2021-04-16 10:34:02 -07:00
Chris Lu 3074e9b428 ensure consistent inode value 2021-04-15 22:42:24 -07:00
Chris Lu 16c0304416 ensure to delete on filer also 2021-04-15 02:29:04 -07:00
Chris Lu 217e0f9066 mount: remove folder recursively 2021-04-15 01:51:10 -07:00
Chris Lu 3e669e6d7b mostly refactoring, add some error handling 2021-04-14 23:33:37 -07:00
Chris Lu 07f712c83f fix typo 2021-04-14 23:21:38 -07:00
Chris Lu 36c79de3f4 fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry 2021-04-14 23:21:24 -07:00
Chris Lu e41766feb6 fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object 2021-04-14 22:38:34 -07:00
Chris Lu 1adc8f86ea lighten up File object 
file.entry only exists when file.isOpen
 2021-04-14 20:49:15 -07:00
Chris Lu c04b7e106f mount: remove entry from Dir object 2021-04-14 20:26:13 -07:00
Chris Lu 3f3268cd1b go fmt 2021-04-14 00:30:16 -07:00
Chris Lu ca0f07a188 move file reader, entryViewCache to file handle 
reduce file object size
 2021-04-14 00:29:58 -07:00
Chris Lu f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu 93f4146ffa properly release the view cache 2021-04-09 12:36:39 -07:00
Chris Lu 6deb647a8f mount: fix possible memory leak 
if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.
 2021-04-08 19:47:31 -07:00
Chris Lu 0f64f5b9c8 mount: add readOnly option 
fix https://github.com/chrislusf/seaweedfs/issues/1961
 2021-04-04 21:40:58 -07:00
Chris Lu bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu 7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu 3cbc40fa48 avoid creating multiple reader 2021-03-22 22:32:47 -07:00
Chris Lu 288369cfc7 mount: release resources when Forget() is called 
address https://github.com/chrislusf/seaweedfs/issues/1929
 2021-03-22 16:22:19 -07:00
Chris Lu df461402cc ensure entry attribute is not nill 
fix https://github.com/chrislusf/seaweedfs/issues/1926
 2021-03-21 22:27:30 -07:00
Chris Lu 69694a17be reverting 7d57664c2d 2021-03-16 02:59:26 -07:00
Chris Lu c00dd5e62e report error if entry not found 2021-03-16 02:55:49 -07:00
Chris Lu 22a07a12d2 [mount] [regression] filer.path option results in empty mount in 2.32 
fix https://github.com/chrislusf/seaweedfs/issues/1906
 2021-03-15 20:46:25 -07:00
Chris Lu 35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu 5fbcaaf37a fullpath() test is not needed 2021-03-12 09:45:25 -08:00
Chris Lu 7d57664c2d mount: internals switch to filer.Entry instead of protobuf 2021-03-12 00:36:38 -08:00
Chris Lu aba47fd9e8 reduce repeated calls to dir.FullPath() 2021-03-11 14:08:20 -08:00
Chris Lu 5ba4b479f8 properly lock file.entry object 
fix https://github.com/chrislusf/seaweedfs/issues/1882
 2021-03-09 23:08:38 -08:00
Chris Lu bcf32591b7 reduce possibility of nil entry 2021-03-02 13:33:56 -08:00
Chris Lu be9c7c21ec reduce possibility of nil file entry 2021-03-02 11:17:17 -08:00
Chris Lu 02e146f3e1 return err 2021-03-02 08:54:18 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu 712b3e9e53 mount: release need to avoid nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1803
 2021-02-13 13:39:37 -08:00
Chris Lu 0604afcda7 mount: remove unnecessary flush on release 2021-02-12 06:01:04 -08:00
Chris Lu a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu 19295600f9 mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes 2021-01-28 15:23:16 -08:00
Chris Lu 990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu cf252fc0cd mount: report error when Flush() 
The error in Release() is not working.

See https://github.com/jaderhs/libfuse/blob/master/FAQ

related to https://github.com/chrislusf/seaweedfs/issues/1765
 2021-01-28 04:46:37 -08:00
Chris Lu b81956bcb5 mount: invalidate kernel cache when mounted to a filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
 2021-01-27 10:28:37 -08:00
Chris Lu 3a1d3d3413 mount: properly invalidate kernel node cache entry 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-26 02:50:53 -08:00
Chris Lu 62ebb917e0 mount: avoid the need to specify different cache directory for different mount 
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
 2021-01-25 09:10:20 -08:00
Chris Lu 17f7c1c43f Merge branch 'mount_from_outside_cluster' 2021-01-24 19:03:02 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 9bcb28a3ea avoid possible nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1754
 2021-01-24 17:19:35 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu 20ef3bb8d4 mount: use direct_io to avoid OS page cache 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-22 22:39:46 -08:00
Chris Lu e439b65e38 fix test 2021-01-18 01:48:00 -08:00
Chris Lu 5a3386e39b Revert "Revert "mount: fake support for socket/block/character/fifo devices"" 
This reverts commit 3d5cb7eb86.
 2021-01-18 01:15:07 -08:00
Chris Lu 67faa56920 Revert "Revert "mount: add more entry type"" 
This reverts commit 4085b79d38.
 2021-01-18 01:14:58 -08:00
Chris Lu 7119e7b491 mount: ensure reading from dirty pages 2021-01-18 01:14:42 -08:00
Chris Lu 4085b79d38 Revert "mount: add more entry type" 
This reverts commit d7d907be27.
 2021-01-18 00:19:19 -08:00
Chris Lu 3d5cb7eb86 Revert "mount: fake support for socket/block/character/fifo devices" 
This reverts commit 61ef2d8658.
 2021-01-18 00:18:57 -08:00
Chris Lu 61ef2d8658 mount: fake support for socket/block/character/fifo devices 
to pass pjdfstest. A distributed file system can not really support these things anyway.
 2021-01-17 23:51:10 -08:00
Chris Lu d7d907be27 mount: add more entry type 2021-01-17 23:46:39 -08:00
Chris Lu a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu 9a50dbcda0 chagned api 2021-01-14 22:18:38 -08:00
Chris Lu d0826c2f62 mount: avoid showing "topics" folder 
fix https://github.com/chrislusf/seaweedfs/issues/1325
 2021-01-14 19:03:15 -08:00
Chris Lu 38d516251e Revert "mount: avoid fatal error" 
This reverts commit a4f0bac9b7.
 2021-01-11 13:50:15 -08:00
Chris Lu a4f0bac9b7 mount: avoid fatal error 2021-01-11 13:26:59 -08:00
Chris Lu 1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu 0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu 3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu 4211601eab set file handle reader to nil 2020-12-08 22:48:18 -08:00
Chris Lu 8e78187a97 add back last read chunk cache to reader and properly close the reader 2020-12-08 22:26:46 -08:00
Chris Lu f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
Chris Lu 1ae108efca add debug message 2020-11-26 10:41:10 -08:00
Chris Lu 0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
Chris Lu 06ee199ef3 ensure entryViewCache is in sync with the chunks 2020-11-03 12:22:19 -08:00
Chris Lu b132cb893f in case some chunks modified earliar are added later 2020-11-03 12:22:19 -08:00
Chris Lu 0ae4d7bfdf lock "addChunks()" 2020-11-03 12:22:19 -08:00
Chris Lu 7d5db73d98 go fmt 2020-11-02 14:21:07 -08:00
Chris Lu a9662d44ac avoid closing chunkSaveErrChan too early 2020-11-02 14:20:38 -08:00
Chris Lu e1423a83dd mount: clear possible leftover entry data in memory 2020-11-01 23:28:56 -08:00
Chris Lu 8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu 4609644658 mount: remove files that may still are open 
related to https://github.com/chrislusf/seaweedfs/issues/1581
 2020-11-01 01:33:26 -08:00
Chris Lu df8d976bb0 refactoring 2020-11-01 01:58:48 -07:00
Chris Lu f7b21973ac mount: add retry for all operations with filer 
fix https://github.com/chrislusf/seaweedfs/issues/1589
 2020-11-01 01:11:08 -07:00
Chris Lu 10f9f1cc26 handle mknod requests 2020-10-30 23:51:32 -07:00
Chris Lu 8826601be1 mount: optional limit for the number of concurrent writers 2020-10-30 21:22:20 -07:00
Chris Lu 19098f2c2d make a copy of the filer.entry for most read operations 2020-10-30 01:23:26 -07:00
Chris Lu 6135cbaa02 clear file.entry 2020-10-30 01:23:26 -07:00
Chris Lu 02dc51b1ed properly clear out the file.entry 2020-10-29 23:31:06 -07:00
Chris Lu 0b68b68ec4 reload entry only when it is a hard link 
fix https://github.com/chrislusf/seaweedfs/issues/1581
 2020-10-29 16:32:45 -07:00
Chris Lu e71463a9eb mount: invalide file cache when metadata is changed 2020-10-25 19:24:15 -07:00
Chris Lu e219c57849 passing full path when assign volume locations 2020-10-25 15:46:29 -07:00
Chris Lu f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu 91fd311f7a fix logic to read entry or not 2020-10-25 10:33:51 -07:00
Chris Lu 10a4a628e9 refresh cached file entry from sync metadata updates 2020-10-25 09:42:50 -07:00
Chris Lu da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Chris Lu 79690b9cd8 revert changes causing wrong data 2020-10-24 20:07:19 -07:00
Chris Lu 6c4fb243a7 remove limits to concurrent writers 2020-10-24 02:05:05 -07:00
Chris Lu ef9c32ea0d adjust compression threshold 2020-10-23 23:05:55 -07:00
Chris Lu 71624858af avoid reusing cached []byte 
this logic is just to ensure 100% correct.

fuse message, data+message hearder > 512 will not be cached.
 2020-10-23 11:31:57 -07:00
Chris Lu f5ee19e0db shortcut for appending operations 2020-10-21 20:44:01 -07:00
Chris Lu b5917bf846 clean up 2020-10-21 19:31:26 -07:00
Chris Lu 9884dfc369 use bytebuffer for interval list 2020-10-21 19:29:51 -07:00
Chris Lu 81cf8d04df minor 2020-10-21 02:42:23 -07:00
Chris Lu d90e7ee3cc adjust log level 2020-10-21 02:17:40 -07:00
Chris Lu 3bf0116de1 mount: less channel waiting 2020-10-21 02:16:21 -07:00
Chris Lu c31b254248 mount: shortcut when there is only one chunk 2020-10-20 23:48:29 -07:00
Chris Lu ad652ffb97 avoid close closed channel 2020-10-20 22:54:34 -07:00
Chris Lu 20442dc1bc avoid extra data copying 2020-10-20 22:54:21 -07:00
Chris Lu 69f0da5d73 mount: pass along replication and collection parameters 2020-10-19 00:12:42 -07:00
Chris Lu c0ab458671 report nil instead of EOF for empty files 
related to https://github.com/chrislusf/seaweedfs/issues/1541
 2020-10-17 11:03:46 -07:00
Chris Lu 09bab17aff mount: avoid "send on closed channel" 2020-10-16 12:54:23 -07:00
Chris Lu 06e6341097 set initial write time to avoid race condition with delayed chunks 2020-10-15 11:08:45 -07:00
Chris Lu d598c47136 log errors 2020-10-15 11:01:43 -07:00
Chris Lu 5039aff310 release the goroutine to read errors 2020-10-15 10:59:18 -07:00
Chris Lu e714c28a02 mount: async write file chunk 2020-10-14 23:28:03 -07:00
Chris Lu 58fa506491 minor 2020-10-13 19:50:22 -07:00
Chris Lu 28d4e1a51b mount: retry for directory listing with filer 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 19:49:52 -07:00
Chris Lu 9b4f7fed14 mount: report filer IO error 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 11:21:13 -07:00
Chris Lu b18f21cce1 mount: fix bound tree with filer.path 
fix https://github.com/chrislusf/seaweedfs/issues/1528
 2020-10-12 21:58:37 -07:00
Chris Lu 7704469d29 mount: outsideContainerClusterMode changed to use volume server publicUrl 2020-10-11 20:42:15 -07:00
Chris Lu 723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu cff8bb6554 return proper error 2020-10-10 15:43:22 -07:00
Chris Lu b067853162 mount: ignore recursion error 
fix https://github.com/chrislusf/seaweedfs/issues/1514

the host OS should manage recursive deletion.
 2020-10-08 21:55:55 -07:00
Chris Lu d77e3c96e9 FUSE: use the mount directory permissions 2020-10-03 13:37:35 -07:00
Chris Lu 474e2b6ac3 add a hard link marker to 16byte + maker, for future extensions 2020-09-27 22:38:30 -07:00
Chris Lu 31fc7bb2e1 refactor 
adjust for faster test
 2020-09-27 10:41:29 -07:00
Chris Lu 043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
Chris Lu 7726965a47 minor 2020-09-24 11:20:12 -07:00
Chris Lu 1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu 2e7c361a0d hardlink deletion factors in hardlink counter 2020-09-24 09:43:52 -07:00
Chris Lu 1295347958 adjust hardlink update 
simplify logic, pass entity content directly to hard link. The "weed mount" handles the logic to calculate hard link counter.
 2020-09-24 09:43:00 -07:00
limd cbca14edc5 mount: fix k8s pvc and os mount directory permission bug 2020-09-24 18:07:16 +08:00
Chris Lu 5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
Chris Lu f0e325b6fc mount: auto created directory follow umask 2020-09-23 02:27:57 -07:00
Chris Lu 0adbb56cc1 rename 2020-09-22 16:24:13 -07:00
Chris Lu 4a1fe4b8e2 add error logs 2020-09-22 09:16:10 -07:00
Chris Lu daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu 387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu 1d56ea24ef fix 2020-09-03 00:08:37 -07:00
Chris Lu 7e1aad0b54 mount: map uid/gid between local and filer 2020-09-03 00:07:22 -07:00
Chris Lu eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu 408e339c53 also delete the manifest chunk itself 2020-08-31 00:16:03 -07:00
Chris Lu 33d8c6c617 change log to warning 2020-08-30 21:01:44 -07:00
Chris Lu e62d7f221f a bit more logs 2020-08-30 21:00:24 -07:00
Chris Lu 22fe4ae573 mount: fix for UrBackup 2020-08-30 21:00:09 -07:00
Chris Lu ae84a9ee2e add logs 2020-08-30 20:59:04 -07:00
Chris Lu a41588279a change log level 5 to 4 2020-08-30 20:12:04 -07:00
Chris Lu f2a8574448 filer and mount deletion resolves manifest chunks also 2020-08-30 02:07:14 -07:00